Skip to content

refactor to work with gardenlinux 0.10.24 - #13

Draft
yeoldegrove wants to merge 16 commits into
mainfrom
feat/bump_python_gardenlinux_lib
Draft

refactor to work with gardenlinux 0.10.24#13
yeoldegrove wants to merge 16 commits into
mainfrom
feat/bump_python_gardenlinux_lib

Conversation

@yeoldegrove

@yeoldegrove yeoldegrove commented Sep 9, 2025

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Code refactoring

glrd/git.py — replace gh/git CLI with python-gardenlinux-lib

Use gardenlinux.github.Client (PyGithub) and gardenlinux.git.Repository
(pygit2) instead of subprocess calls to gh and git. Both are already
pulled in transitively by the gardenlinux dependency.

  • get_github_releases: Client().get_repo(...).get_releases()
  • get_git_commit_from_tag: get_git_ref() with annotated-tag dereferencing
  • get_git_commit_at_time: Repository.checkout_repo() + pygit2 walk
  • cleanup_temp_repo: dereference _repo_instance before rmtree

Auth change: GITHUB_TOKEN is now required for GitHub API calls instead
of gh auth login. Remove gh CLI install from Containerfile.

release.py (renamed from models.py) — single source of truth

  • Harden Version for 'next' sentinel; make to_dict faithful
  • Wire as SSoT in query.py (filter, sort, latest, active/archived) and
    manage.py (create, parse_release_name, ensure_isodate_and_timestamp)
  • Add Release.default_name() and Release.github_release_url()

manage.py — remove duplicates

Remove functions now imported from git.py, validation.py, and s3.py`.
Remove unused imports.

schema.py — unify schema_v1.py and schema_v2.py

Generate SCHEMA_V1 and SCHEMA_V2 from shared builder functions.
Reduces of duplicated definitions. Delete schema_v1.py and
schema_v2.py.

Other refactors

  • Extract validation functions to glrd/validation.py
  • Extract S3 operations to glrd/s3.py
  • Extract shared helpers into glrd/util.py (V2_SCHEMA_THRESHOLD,
    split_releases_by_type, run_subprocess, parse_isodatetime, fatal_error,
    resolve_flavors, merge_input_data)
  • Bump python-gardenlinux-lib dependency from 0.6.0 to 0.10.24.
  • Fix circular import: move merge_input_data from manage to util
  • Fix filter_releases crash on unknown --type values
  • Add --no-flavors flag and GLRD_SKIP_FLAVORS env var
  • Add --input-type/--input-url/--input-file-prefix/--query-input-format
    to glrd-manage for offline/custom-source queries
  • Fix deprecated datetime.utcfromtimestamp() usage throughout

glrd/util.py + glrd/query.py — split CONTAINER_REGISTRY by release type

  • Rename DEFAULTS['CONTAINER_REGISTRY']DEFAULTS['CONTAINER_REGISTRY_MINOR']
    (ghcr.io/gardenlinux/gardenlinux)
  • Add DEFAULTS['CONTAINER_REGISTRY_NIGHTLY'] (ghcr.io/gardenlinux/nightly)
  • Add get_container_registry(release_type) helper in query.py that returns
    the nightly registry for nightly releases and the minor registry otherwise
  • Thread release_type through prepare_oci_flavor_url (new 4th param) and
    its call site in format_flavors_with_urls
  • Update get_oci_url to use get_container_registry(release.get('type'))

Tests

  • Make full test suite fully offline (was hanging on network before) via autouse conftest fixture
  • Add tests/test_git.py: unit tests for all git.py functions, mocking
    gardenlinux.github.Client and using a local pygit2 bare-repo fixture
  • Add tests/test_release_model.py: unit tests for the release model
  • Add offline CLI integration tests (active/archived/latest/version
    filter, no-flavors, unknown-type guard)
  • Add tests/test_query_oci.py: unit tests covering get_container_registry,
    get_oci_url, and prepare_oci_flavor_url for both container and bare flavors
  • Add OCI integration tests to test_integration.py: nightly registry selection,
    minor registry selection, and cross-type registry difference

Documentation

  • Replace monolithic docs/overview/README.md with a full Diataxis
    structure: reference/index.md, reference/cli.md, reference/schema.md,
    explanation/release-lifecycle.md, how-to/index.md, how-to/run-glrd.md,
    how-to/query-releases.md, how-to/manage-releases.md
  • Replace gh CLI prerequisite in run-glrd.md with GITHUB_TOKEN
    prerequisite; add GITHUB_TOKEN callout to manage-releases.md

Fixes

Fixes #16
Fixes #14
Fixes #4

@yeoldegrove yeoldegrove self-assigned this Sep 9, 2025
@yeoldegrove
yeoldegrove force-pushed the feat/bump_python_gardenlinux_lib branch 2 times, most recently from e5121c4 to f911488 Compare September 9, 2025 13:42
@yeoldegrove
yeoldegrove force-pushed the feat/bump_python_gardenlinux_lib branch from f911488 to 28a7960 Compare September 23, 2025 11:37
@yeoldegrove yeoldegrove linked an issue Sep 23, 2025 that may be closed by this pull request
@yeoldegrove
yeoldegrove force-pushed the feat/bump_python_gardenlinux_lib branch from 28a7960 to 16cd1c4 Compare October 8, 2025 10:35
@yeoldegrove yeoldegrove changed the title refactor to work with gardenlinux 0.9.1 refactor to work with gardenlinux 0.10.0 Oct 8, 2025
@yeoldegrove
yeoldegrove force-pushed the feat/bump_python_gardenlinux_lib branch from 16cd1c4 to 4e3988d Compare October 8, 2025 11:27
@yeoldegrove
yeoldegrove force-pushed the feat/bump_python_gardenlinux_lib branch from 4e3988d to 7e02f13 Compare April 9, 2026 10:14
@yeoldegrove yeoldegrove changed the title refactor to work with gardenlinux 0.10.0 refactor to work with gardenlinux 0.10.20 Apr 9, 2026
… schema.py

Reduces ~580 lines of duplicated schema definitions by generating both
SCHEMA_V1 and SCHEMA_V2 from shared builder functions. The only difference
between v1 and v2 is that v2 requires a 'patch' field in version objects
for minor, nightly, and dev release types.

- Create glrd/schema.py with factory functions for generating schemas
- Update manage.py to import from glrd.schema instead of the old files
- All schema validation tests pass (7/7)
- Add missing ERROR_CODES entries: output_error, s3_output_error,
  input_parameter_missing, input_parameter_error, subprocess_output_missing
- Add fatal_error() utility function for centralized error handling
- Fix deprecated datetime.utcfromtimestamp() usage in util.py and manage.py
  (use datetime.fromtimestamp with timezone.utc instead)
- Add timezone import to both util.py and manage.py
Add commonly used utility functions:
- V2_SCHEMA_THRESHOLD constant (2017) for v1/v2 schema decisions
- uses_patch_version() helper function
- split_releases_by_type() to split releases by type into a dict
- run_subprocess() for standardized subprocess execution
- parse_isodatetime() for consistent datetime parsing with error handling

All schema tests pass.
Replace duplicated release type filtering logic (~15 lines repeated 3 times)
with calls to the shared split_releases_by_type() utility function from util.py.
This eliminates duplicate code and ensures consistent behavior.
- Remove unused import in schema.py
- Fix line length issues in schema.py
- Add proper blank line after function in util.py
- Add proper type hints for parse_isodatetime
- Format all modified files with black
Phase 1 - Dataclasses:
- Add glrd/models.py with ReleaseType enum, Version, LifecyclePhase,
  Lifecycle, GitInfo, Release, ReleaseCollection, parse_release_name
- Centralizes the 'major >= 2017' version threshold logic in Version class

Phase 2 - S3 module:
- Extract S3 operations to glrd/s3.py
- Import s3 functions in manage.py for backward compatibility
- Functions: create_s3_bucket, upload_to_s3, download_from_s3,
  merge_existing_s3_data, download_all_s3_files, upload_all_local_files

All schema validation tests pass.
Phase 2 - Git module:
- Extract Git operations from manage.py to glrd/git.py
- Functions: get_github_releases, get_git_commit_from_tag,
  get_git_commit_at_time, get_garden_version_for_date,
  create_initial_releases, create_initial_nightly_releases, cleanup_temp_repo
- Fix date increment logic in create_initial_nightly_releases (use timedelta)
- Import git functions in manage.py

All schema tests pass.
Phase 2 - Validation module:
- Extract validation functions from manage.py to glrd/validation.py
- Functions: validate_input_version_format, get_schema_for_release,
  validate_release_data, validate_all_releases
- Use V2_SCHEMA_THRESHOLD constant instead of hardcoded 2017
- Import validation functions in manage.py

All schema tests pass.
- Remove unused datetime import from models.py
- Remove unused major_version variable in git.py
- Fix trailing whitespace in git.py
- Format code with black

All tests passing.
- Change from 'deepdiff import DeepDiff' to 'deepdiff.diff import DeepDiff'
  to fix lint warning about incorrect module import

Note: manage.py still contains duplicate function definitions that are
imported from new modules (s3.py, git.py, validation.py). These duplicates
are functionally harmless as Python uses the imported versions, but they
trigger F811 lint warnings. Removing them automatically breaks the file due
to complex interdependencies. This is acceptable technical debt that can
be addressed incrementally.

All 29 tests passing.
Clean up manage.py by removing 522 lines of duplicate function definitions
that are now imported from other modules:

Duplicate functions removed:
- validate_input_version_format, get_schema_for_release (from validation.py)
- cleanup_temp_repo, get_github_releases, get_git_commit_from_tag,
  get_git_commit_at_time, get_garden_version_for_date,
  create_initial_releases, create_initial_nightly_releases (from git.py)
- validate_release_data, validate_all_releases (from validation.py)
- download_all_s3_files, upload_all_local_files (from s3.py)

Unused imports removed:
- fnmatch, shutil, subprocess, timedelta, timezone
- jsonschema.validate, jsonschema.ValidationError
- SCHEMA_V1, SCHEMA_V2
- extract_version_data, NoAliasDumper
- save_output_file alias

File size reduced from 1,911 to 1,364 lines (-547 lines, -29%)
Format with black to fix spacing issues.

All 29 tests passing. Lint clean (no errors or warnings).
@yeoldegrove
yeoldegrove force-pushed the feat/bump_python_gardenlinux_lib branch 2 times, most recently from d0fe837 to 2eecb72 Compare July 22, 2026 05:06
@yeoldegrove yeoldegrove changed the title refactor to work with gardenlinux 0.10.20 refactor to work with gardenlinux 0.10.21 Jul 22, 2026
@yeoldegrove
yeoldegrove force-pushed the feat/bump_python_gardenlinux_lib branch 2 times, most recently from 8bf6ea1 to ec58af6 Compare July 22, 2026 08:52
@gardenlinux-docs-bot

Copy link
Copy Markdown

📚 Documentation Preview PR: gardenlinux/docs#132
😎 Deploy Preview: https://deploy-preview-132--gardenlinux-docs.netlify.app

A PR has been created/updated to preview the documentation changes from this PR.

@yeoldegrove
yeoldegrove force-pushed the feat/bump_python_gardenlinux_lib branch 2 times, most recently from 168cc1c to f7d9722 Compare August 25, 2026 06:40
…test suite fully offline

- Bump python-gardenlinux-lib dependency from 0.10.20 to 0.10.24
- Rename models.py -> release.py; harden Version for 'next' sentinel;
  make to_dict faithful (2135 real releases round-trip with 0 mismatches)
- Wire release.py as single source of truth in query.py (filter, sort,
  latest, active/archived) and manage.py (create builds Release object,
  parse_release_name delegates to model, ensure_isodate_and_timestamp
  delegates to LifecyclePhase.ensure_complete)
- Add Release.default_name() and Release.github_release_url() so name
  and URL generation are centralized in the model
- Fix circular import: move merge_input_data from manage to util
- Remove duplicate S3 functions from manage.py (save_output_file,
  create_s3_bucket, upload_to_s3, download_from_s3, merge_existing_s3_data);
  import from glrd.s3 throughout
- Delete unused schema_v1.py and schema_v2.py
- Wire previously dead --s3-create-bucket flag in handle_releases
- Fix filter_releases crash on unknown --type values (restore prior behavior
  of silently ignoring unknown types instead of raising ValueError)
- Fix dead pass branch in diff_releases to log added/removed items
- Add resolve_flavors() to util.py; add --no-flavors flag and
  GLRD_SKIP_FLAVORS env var to skip network-dependent flavor lookup
- Skip git commit lookup for next/major release types (they never store it)
- Add --input-type/--input-url/--input-file-prefix/--query-input-format to
  glrd-manage so update/delete queries can run offline or against custom sources
- Make full test suite offline (was hanging indefinitely; now 83 tests in ~39s):
  autouse conftest fixture, offline query injection in test helpers
- Add 22 new tests: 21 unit tests for release model + 6 offline CLI integration
  tests (active/archived/latest/version filter, no-flavors, unknown-type guard)
- Fix test_schema_validation.py import (glrd.validation, not glrd.manage)
- Update README with current --help output and offline testing guidance
- Net -375 lines; flake8 + black clean

Signed-off-by: Eike Waldt <waldt@b1-systems.de>
On-behalf-of: SAP <eike.waldt@sap.com>
Assisted-by: Kilo:claude-opus-4-8
Assisted-by: Kilo:claude-sonnet-4-6
@yeoldegrove
yeoldegrove force-pushed the feat/bump_python_gardenlinux_lib branch from f7d9722 to 577b932 Compare August 25, 2026 07:07
@yeoldegrove yeoldegrove changed the title refactor to work with gardenlinux 0.10.21 refactor to work with gardenlinux 0.10.24 Aug 25, 2026
Replace the monolithic docs/overview/README.md with a
proper Diataxis structure:

- docs/reference/index.md      → hub: reference/supporting_tools/glrd/index.md
- docs/reference/cli.md        → hub: reference/supporting_tools/glrd/cli.md
- docs/reference/schema.md     → hub: reference/supporting_tools/glrd/schema.md
- docs/explanation/release-lifecycle.md → hub: explanation/glrd-release-lifecycle.md
- docs/how-to/index.md         → hub: how-to/releases/glrd/index.md
- docs/how-to/run-glrd.md      → hub: how-to/releases/glrd/run-glrd.md
- docs/how-to/query-releases.md → hub: how-to/releases/glrd/query-releases.md
- docs/how-to/manage-releases.md → hub: how-to/releases/glrd/manage-releases.md

Signed-off-by: Eike Waldt <waldt@b1-systems.de>
On-behalf-of: SAP <eike.waldt@sap.com>
Assisted-by: Kilo:claude-opus-4-8
Assisted-by: Kilo:claude-sonnet-4-6
Use gardenlinux.github.Client (PyGithub) and gardenlinux.git.Repository
(pygit2) instead of subprocess calls to gh and git. Both are already
pulled in transitively by the gardenlinux dependency.

- get_github_releases: Client().get_repo(...).get_releases()
- get_git_commit_from_tag: get_git_ref() with annotated-tag dereferencing
- get_git_commit_at_time: Repository.checkout_repo() + pygit2 walk
- cleanup_temp_repo: dereference _repo_instance before rmtree

Auth change: GITHUB_TOKEN is now required for GitHub API calls instead
of gh auth login. Add GITHUB_TOKEN prerequisites to run-glrd.md and
manage-releases.md. Remove gh CLI install from Containerfile.

Add tests/test_git.py with 14 unit tests covering all changed functions.

Signed-off-by: Eike Waldt <waldt@b1-systems.de>
On-behalf-of: SAP <eike.waldt@sap.com>
Assisted-by: Kilo:claude-opus-4-8
Assisted-by: Kilo:claude-sonnet-4-6
- Rename DEFAULTS['CONTAINER_REGISTRY'] → DEFAULTS['CONTAINER_REGISTRY_MINOR']
  (ghcr.io/gardenlinux/gardenlinux)
- Add DEFAULTS['CONTAINER_REGISTRY_NIGHTLY'] (ghcr.io/gardenlinux/nightly)
- Add get_container_registry(release_type) helper in query.py that returns
  the nightly registry for nightly releases and the minor registry otherwise
- Thread release_type through prepare_oci_flavor_url (new 4th param) and
  its call site in format_flavors_with_urls
- Update get_oci_url to use get_container_registry(release.get('type'))
- Add tests/test_query_oci.py: 11 unit tests covering get_container_registry,
  get_oci_url, and prepare_oci_flavor_url for both container and bare flavors
- Add OCI integration tests to test_integration.py: nightly/minor registry
  selection and cross-type registry difference

Fixes #16

Signed-off-by: Eike Waldt <waldt@b1-systems.de>
On-behalf-of: SAP <eike.waldt@sap.com>
Assisted-by: Kilo:claude-opus-4-7
Assisted-by: Kilo:claude-sonnet-4-6
@yeoldegrove
yeoldegrove force-pushed the feat/bump_python_gardenlinux_lib branch from 577b932 to c7d8fbe Compare August 25, 2026 07:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

glrd: nightly container image tags use wrong url glrd: update documentation for semantic versioning refactoring of GLRD codebase

1 participant